Skip to content

Add logical range partitioning representation#22777

Open
gene-bordegaray wants to merge 1 commit into
apache:mainfrom
gene-bordegaray:gene.bordegaray/2026/06/logical-source-partitioning
Open

Add logical range partitioning representation#22777
gene-bordegaray wants to merge 1 commit into
apache:mainfrom
gene-bordegaray:gene.bordegaray/2026/06/logical-source-partitioning

Conversation

@gene-bordegaray
Copy link
Copy Markdown
Contributor

@gene-bordegaray gene-bordegaray commented Jun 5, 2026 •
edited
Loading

Which issue does this PR close?

Rationale for this change

Declared scan output partitioning should use logical partitioning metadata, not physical partitioning types. This adds logical range partitioning so range-partitioned sources can declare their layout at the logical layer.

What changes are included in this PR?

  • Add logical Partitioning::Range and RangePartitioning.
  • Move SplitPoint and shared split-point validation to datafusion-common.
  • Wire logical range partitioning through expression traversal, rewrites, and display.
  • Keep planning, logical proto, and Substrait support explicitly unsupported for now.

Are these changes tested?

Yes. Unit tests added

Are there any user-facing changes?

Yes. This adds public logical range partitioning API. No breaking API changes.

@gene-bordegaray gene-bordegaray changed the title Add logical range partitioning representation [WIP] Add logical range partitioning representation Jun 5, 2026
@github-actions github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate substrait Changes to the substrait crate common Related to common crate proto Related to proto crate labels Jun 5, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 5, 2026

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details 
     Cloning apache/main
    Building datafusion v53.1.0 (current)
       Built [ 103.808s] (current)
     Parsing datafusion v53.1.0 (current)
      Parsed [   0.037s] (current)
    Building datafusion v53.1.0 (baseline)
       Built [ 100.980s] (baseline)
     Parsing datafusion v53.1.0 (baseline)
      Parsed [   0.039s] (baseline)
    Checking datafusion v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.962s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 207.764s] datafusion
    Building datafusion-common v53.1.0 (current)
       Built [  34.680s] (current)
     Parsing datafusion-common v53.1.0 (current)
      Parsed [   0.065s] (current)
    Building datafusion-common v53.1.0 (baseline)
       Built [  34.111s] (baseline)
     Parsing datafusion-common v53.1.0 (baseline)
      Parsed [   0.063s] (baseline)
    Checking datafusion-common v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   1.031s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  71.215s] datafusion-common
    Building datafusion-expr v53.1.0 (current)
       Built [  26.907s] (current)
     Parsing datafusion-expr v53.1.0 (current)
      Parsed [   0.078s] (current)
    Building datafusion-expr v53.1.0 (baseline)
       Built [  27.096s] (baseline)
     Parsing datafusion-expr v53.1.0 (baseline)
      Parsed [   0.080s] (baseline)
    Checking datafusion-expr v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   1.774s] 223 checks: 221 pass, 1 fail, 1 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_variant_added.ron

Failed in:
  variant Partitioning:Range in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:4450
  variant Partitioning:Range in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:4450

--- warning partial_ord_enum_variants_reordered: enum variants reordered in #[derive(PartialOrd)] enum ---

Description:
A public enum that derives PartialOrd had its variants reordered. #[derive(PartialOrd)] uses the enum variant order to set the enum's ordering behavior, so this change may break downstream code that relies on the previous order.
        ref: https://doc.rust-lang.org/std/cmp/trait.PartialOrd.html#derivable
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/partial_ord_enum_variants_reordered.ron

Failed in:
  Partitioning::DistributeBy moved from position 3 to 4, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:4452
  Partitioning::DistributeBy moved from position 3 to 4, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:4452

     Summary semver requires new major version: 1 major and 0 minor checks failed
     Warning produced 1 major and 0 minor level warnings
    Finished [  57.611s] datafusion-expr
    Building datafusion-physical-expr v53.1.0 (current)
       Built [  28.895s] (current)
     Parsing datafusion-physical-expr v53.1.0 (current)
      Parsed [   0.051s] (current)
    Building datafusion-physical-expr v53.1.0 (baseline)
       Built [  28.985s] (baseline)
     Parsing datafusion-physical-expr v53.1.0 (baseline)
      Parsed [   0.054s] (baseline)
    Checking datafusion-physical-expr v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.541s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure struct_missing: pub struct removed or renamed ---

Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/struct_missing.ron

Failed in:
  struct datafusion_physical_expr::SplitPoint, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/a5cdb2ad55310fc69c38f794c984085911be093f/datafusion/physical-expr/src/partitioning.rs:225

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  59.682s] datafusion-physical-expr
    Building datafusion-proto v53.1.0 (current)
       Built [  59.909s] (current)
     Parsing datafusion-proto v53.1.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-proto v53.1.0 (baseline)
       Built [  60.061s] (baseline)
     Parsing datafusion-proto v53.1.0 (baseline)
      Parsed [   0.022s] (baseline)
    Checking datafusion-proto v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.375s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 122.088s] datafusion-proto
    Building datafusion-substrait v53.1.0 (current)
       Built [ 343.611s] (current)
     Parsing datafusion-substrait v53.1.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-substrait v53.1.0 (baseline)
       Built [ 344.503s] (baseline)
     Parsing datafusion-substrait v53.1.0 (baseline)
      Parsed [   0.021s] (baseline)
    Checking datafusion-substrait v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.303s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 690.609s] datafusion-substrait

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jun 5, 2026
@gene-bordegaray gene-bordegaray changed the title [WIP] Add logical range partitioning representation Add logical range partitioning representation Jun 5, 2026
@gene-bordegaray
Copy link
Copy Markdown
Contributor Author

gene-bordegaray commented Jun 5, 2026

cc: @gabotechs @stuhood

@gene-bordegaray gene-bordegaray marked this pull request as ready for review June 5, 2026 18:35
stuhood
stuhood approved these changes Jun 5, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Comment thread datafusion/core/src/physical_planner.rs
Comment on lines +1268 to +1270
return not_impl_err!(
"Physical plan does not support Range repartitioning"
);
Copy link
Copy Markdown
Contributor

@stuhood stuhood Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a TODO, right? Should it point at a ticket?

Copy link
Copy Markdown
Contributor Author

@gene-bordegaray gene-bordegaray Jun 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes good catch I can add the epic: #22395

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@stuhood stuhood stuhood approved these changes

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

auto detected api change Auto detected API change common Related to common crate core Core DataFusion crate logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates proto Related to proto crate substrait Changes to the substrait crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add logical range partitioning representation

2 participants

@gene-bordegaray @stuhood